Aiming at the issue of ineffective fusion of multi-modal features of indoor scene semantic segmentation using RGB-D， a network named APFNet （Attention mechanism and Pyramid Fusion Network） was proposed， in which attention mechanism fusion module and pyramid fusion module were designed. To fully use the complementarity of the RGB features and the Depth features， the attention allocation weights of these two kinds of features were respectively extracted by the attention mechanism fusion module， making the network focus more on the multi-modal feature domain with more information content. Local and global information were fused by pyramid fusion module with four different scales of pyramid features， thus scene context was extracted and segmentation accuracies of object edges and small-scale objects were improved. By integrating these two fusion modules into a three-branch “encoder-decoder” network， an “end-to-end” output was realized. Comarative experiments were implemented with the state-of-the-art methods， such as multi-level RGB-D residual feature Fusion network （RDF-152）， Attention Complementary features Network （ACNet） and Spatial information Guided convolution Network （SGNet） on the SUN RGB-D and NYU Depth v2 datasets. Compared with the best-performing method RDF-152， when the layer number of the encoder network was reduced from 152 to 50， the Pixel Accuracy （PA）， Mean Pixel Accuracy （MPA）， and Mean Intersection over Union （MIoU） of APFNet were respectively increased by 0.4， 1.1 and 3.2 percentage points. The semantic segmentation accuracies for small-scale objects such as pillows and photos， and large-scale objects such as boards and ceilings were increased by 0.9 to 3.4 and 12.4 to 18 percentage points respectively. The results show that the proposed APFNet has some advantages in dealing with the semantic segmentation of indoor scenes.